1. Technical Field
The invention is related to the generation and rendering of video, and more particularly to a system and process for generating and rendering an interactive viewpoint video in which a user can watch a dynamic scene while changing the viewpoint at will.
2. Background Art
For several years now, viewers of TV commercials and feature films have been seeing the “freeze frame” effect used to create the illusion of stopping time and changing the camera viewpoint. The earliest commercials were produced by using a film-based system, which rapidly jumped between different still cameras arrayed along a rail to give the illusion of moving through a frozen slice of time.
When it first appeared, the effect was fresh and looked spectacular, and soon it was being emulated in many productions, the most famous of which is probably the “bullet time” effects seen in the movie entitled “The Matrix”. Unfortunately, this effect is a one-time, pre-planned affair. The viewpoint trajectory is planned ahead of time, and many man hours are expended to produce the desired interpolated views. Newer systems are based on video camera arrays, but still rely on having many cameras to avoid software view interpolation.
Thus, existing systems would not allow a user to interactively change to any desired viewpoint while watching a dynamic image-based scene. Most of the work on image-based rendering (IBR) in the past involves rendering static scenes, with two of the best-known techniques being Light Field Rendering [12] and the Lumigraph [8]. Their success in high quality rendering stems from the use of a large number of sampled images and has inspired a large body of work in the field. One exciting potential extension of this groundbreaking work involves interactively controlling viewpoint while watching a video. The ability of a user to interactively control the viewpoint of a video enhances the viewing experience considerably, enabling such diverse applications as new viewpoint instant replays, changing the point of view in dramas, and creating “freeze frame” visual effects at will.
However, extending IBR to dynamic scenes is not trivial because of the difficulty (and cost) of synchronizing so many cameras as well as acquiring and storing the images. Not only are there significant hurdles to overcome in capturing, representing, and rendering dynamic scenes from multiple points of view, but being able to do this interactively provides a significant further complication. To date attempts to realize this goal have not been very satisfactory.
In regard to the video-based rendering aspects of an interactive viewpoint video system, one of the earliest attempts at capturing and rendering dynamic scenes was Kanade et al's Virtualized Reality system [11], which involved 51 cameras arranged around a 5-meter geodesic dome. The resolution of each camera is 512×512 and the capture rate is 30 fps. They extract a global surface representation at each time frame, using a form of voxel coloring [15] based on the scene flow equation [18]. Unfortunately, the results look unrealistic because of low resolution, matching errors, and improper handling of object boundaries.
Carranza et al. [3] used seven synchronized cameras distributed around a room looking towards its center to capture 3D human motion. Each camera is at CIF resolution (320×240) and captures at 15 fps. They use a 3D human model as a prior to compute 3D shape at each time frame.
Yang et al. [21] designed an 8×8 grid of cameras (each 320×240) for capturing a dynamic scene. Instead of storing and rendering the data, they transmit only the rays necessary to compose the desired virtual view. In their system, the cameras are not genlocked; instead, they rely on internal clocks across six PCs. The camera capture rate is 15 fps, and the interactive viewing rate is 18 fps.
As a proof of concept for storing dynamic light fields, Wilburn et al. [20] demonstrated that it is possible to synchronize six cameras (640×480 at 30 fps), and compress and store all the image data in real time. They have since hooked up 128 cameras.
A lot of images are required for realistic rendering if the scene geometry is either unknown or known to only a rough approximation. If geometry is known accurately, it is possible to reduce the requirement for images substantially [8]. One practical way of extracting the scene geometry is through stereo, and a lot of stereo algorithms have been proposed for static scenes [14]. However, there have been a few attempts at employing stereo techniques with dynamic scenes. As part of the Virtualized Reality work [11], Vedula et al. [18] proposed an algorithm for extracting 3D motion (i.e., correspondence between scene shape across time) using 2D optical flow and 3D scene shape. In their approach, they use a voting scheme similar to voxel coloring [15], where the measure used is how well a hypothesized voxel location fits the 3D flow equation.
Zhang and Kambhamettu [22] also integrated 3D scene flow and structure in their framework. Their 3D affine motion model is used locally, with spatial regularization, and discontinuities are preserved using color segmentation. Tao et al. [17] assume the scene is piecewise planar. They also assume constant velocity for each planar patch in order to constrain the dynamic depth map estimation.
In a more ambitious effort, Carceroni and Kutulakos [2] recover piecewise continuous geometry and reflectance (Phong model) under non-rigid motion with known lighting positions. They discretize the space into surface elements (“surfers”), and perform a search over location, orientation, and reflectance parameter to maximize agreement with the observed images.
In an interesting twist to the conventional local windowmatching, Zhang et al. [23] use matching windows that straddle space and time. The advantage of this method is that there is less dependence on brightness constancy over time.
Active rangefinding techniques have also been applied to moving scenes. Hall-Holt and Rusinkiewicz [9] use projected boundary-coded stripe patterns that vary over time. There is also a commercial system on the market called ZCam™ manufactured by 3DV Systems of Israel, which is a range sensing video camera add-on used in conjunction with a broadcast video camera. However, it is an expensive system, and provides single viewpoint depth only, which makes it less suitable for multiple view-point video.
However, despite all the advances in stereo and image-based rendering, it is still very difficult to interactively render high-quality, high resolution views of dynamic scenes. The present invention tackles this problem in a cost efficient manner.
It is noted that in the preceding paragraphs, as well as in the remainder of this specification, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [1]” or simply “[1]”. Multiple references will be identified by a pair of brackets containing more than one designator, for example, [2, 3]. A listing of references including the publications corresponding to each designator can be found at the end of the Detailed Description section.